{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c8b91583-ef54-4974-8a23-a62c15ddf01c",
   "metadata": {},
   "source": [
    "# Anachronism Test with Guidance Server\n",
    "\n",
    "In this notebook, we will re-implement the Anachronism example notebook, but use Guidance Server as the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f60fe8b-559e-484c-a2a1-4b2bcdb19e62",
   "metadata": {},
   "source": [
    "# The Server Process\n",
    "\n",
    "First, we need to start the Guidance server itself. This needs to run in a separate process, so we have to write out a simple script file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3c53d7b1-6cc0-49c2-8a68-81672a4c6ef2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting guidance_server.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile guidance_server.py\n",
    "\n",
    "import logging\n",
    "import os\n",
    "import pathlib\n",
    "\n",
    "import guidance\n",
    "\n",
    "_logger = logging.getLogger(__file__)\n",
    "_logger.setLevel(logging.INFO)\n",
    "\n",
    "\n",
    "def main():\n",
    "    win_home = target_model_path = os.getenv(\"MODEL_PATH\")\n",
    "    _logger.info(f\"Attempting to load {target_model_path}\")\n",
    "\n",
    "    lm = guidance.models.LlamaCpp(target_model_path, n_gpu_layers=-1)\n",
    "    _logger.info(\"Model loaded\")\n",
    "\n",
    "    server = guidance.Server(lm, api_key=\"SDFSDF\")\n",
    "    _logger.info(\"Server object created\")\n",
    "    server.run(port=8392)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b422c36-d3ec-4e44-a825-1e661edd1074",
   "metadata": {},
   "source": [
    "Now we can run this in a separate python process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c65b4c1a-e1d8-4b65-8a7a-7f5d5572ff46",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Server PID: 2026\n"
     ]
    }
   ],
   "source": [
    "import subprocess\n",
    "\n",
    "server_process = subprocess.Popen([\"python\", \"./guidance_server.py\"])\n",
    "\n",
    "print(f\"Server PID: {server_process.pid}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c795bfe5-e026-48ec-b1c4-27849c2a845d",
   "metadata": {},
   "source": [
    "## Creating the 'client' model\n",
    "\n",
    "With the server running, we create a model, giving it the URI of the Guidance server endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c8464bf9-f9bd-47fa-9166-68265c44c3bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import guidance\n",
    "\n",
    "lm = guidance.models.Model(\"http://localhost:8392\", api_key=\"SDFSDF\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d742d761-1268-4a4a-8942-3e2aa2d43255",
   "metadata": {},
   "source": [
    "We can run a trial generation. Note that you may have to wait for a few minutes for the server process to be ready (you will get a 'Connection Refused' error if it is not):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fe9c2870-99bf-42d9-95a2-cef0f4cf2471",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; vertical-align: middle; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>Tell me a joke.<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>I</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>&#x27;</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>m</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> not</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> sure</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> if</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> I</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>&#x27;</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>m</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> in</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> the</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> m</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ood</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> for</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> a</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> jo</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ke</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>.</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span></pre>"
      ],
      "text/plain": [
       "<guidance.models._model.Model at 0x7fb5df8a2a90>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm + \"Tell me a joke.\" + guidance.gen(\"simple_joke\", max_tokens=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b15ff7-a319-48c5-82dd-f5fddfaa5e1b",
   "metadata": {},
   "source": [
    "## Loading the data\n",
    "\n",
    "Next, we load the anachronism data as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5fc8fda6-38b5-4c34-8cea-40d422f6e828",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-02-05 09:23:53.849454: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-02-05 09:23:53.849571: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-02-05 09:23:53.855983: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-02-05 09:23:53.925994: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-02-05 09:23:55.689865: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 46 data items\n"
     ]
    }
   ],
   "source": [
    "import datasets\n",
    "\n",
    "# load the data\n",
    "data = datasets.load_dataset('bigbench', 'anachronisms')\n",
    "inputs = [x.split('\\n')[0] for x in data['validation']['inputs']]\n",
    "labels = [x[0] for x in data['validation']['targets']]\n",
    "\n",
    "print(f\"Loaded {len(labels)} data items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea07d211-788c-4060-bc85-0c64d95303ba",
   "metadata": {},
   "source": [
    "## A Guidance Program\n",
    "\n",
    "We can then define a guidance program which creates a simple prompt and sends it to the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "17c9a133-7df6-4194-9421-b4c48cc329a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "@guidance\n",
    "def anachronism_query(llm, query, examples):\n",
    "    prompt_string = \"\"\"Given a sentence tell me whether it contains an anachronism (i.e. whether it could have happened or\n",
    "not based on the time periods associated with the entities).\n",
    "\n",
    "Here are some examples:\n",
    "\"\"\"\n",
    "    for ex in examples:\n",
    "        prompt_string += f\"Sentence: { ex['input'] }\" + \"\\n\"\n",
    "        prompt_string += \"Entities and dates:\\n\"\n",
    "        for en in ex['entities']:\n",
    "            prompt_string += f\"{en['entity']} : {en['time']}\" + \"\\n\"\n",
    "        prompt_string += f\"Reasoning: {ex['reasoning']}\" + \"\\n\"\n",
    "        prompt_string += f\"Anachronism: {ex['answer']}\" + \"\\n\"\n",
    "\n",
    "    llm += f'''{prompt_string}\n",
    "Now determine whether the following is an anachronism:\n",
    "    \n",
    "Sentence: { query }\n",
    "Entities and dates:\n",
    "{ guidance.gen(name=\"entities\", max_tokens=100, stop=\"Reason\") }'''\n",
    "    llm += \"Reasoning :\"\n",
    "    llm += guidance.gen(name=\"reason\", max_tokens=100, stop=\"\\n\")\n",
    "    llm += f'''\\nAnachronism: { guidance.select([\"Yes\", \"No\"], name=\"answer\") }'''\n",
    "    return llm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61f64f00-a1c0-49e4-8c29-88d420dd39c5",
   "metadata": {},
   "source": [
    "Now, call the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "06bee9ca-21b9-49d4-adf9-8228c83e6fde",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; vertical-align: middle; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>Given a sentence tell me whether it contains an anachronism (i.e. whether it could have happened or\n",
       "not based on the time periods associated with the entities).\n",
       "\n",
       "Here are some examples:\n",
       "Sentence: I wrote about shakespeare\n",
       "Entities and dates:\n",
       "I : present\n",
       "Shakespeare : 16th century\n",
       "Reasoning: I can write about Shakespeare because he lived in the past with respect to me.\n",
       "Anachronism: No\n",
       "Sentence: Shakespeare wrote about me\n",
       "Entities and dates:\n",
       "Shakespeare : 16th century\n",
       "I : present\n",
       "Reasoning: Shakespeare cannot have written about me, because he died before I was born\n",
       "Anachronism: Yes\n",
       "\n",
       "Now determine whether the following is an anachronism:\n",
       "\n",
       "Sentence: The T-Rex bit my dog\n",
       "Entities and dates:<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>T</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>-</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>Rex</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> :</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> </span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>6</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>5</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> million</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> years</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> ago</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>D</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>og</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> :</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> present</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span>Reasoning :<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> The</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> T</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>-</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>R</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ex</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> is</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> ext</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>inct</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>,</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> so</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> it</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> cannot</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> b</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ite</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> my</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> dog</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>.</span>\n",
       "Anachronism:<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> Yes</span></pre>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "entities:\n",
      "T-Rex : 65 million years ago\n",
      "Dog : present\n",
      "\n",
      "reasoning:  The T-Rex is extinct, so it cannot bite my dog.\n",
      "answer: Yes\n"
     ]
    }
   ],
   "source": [
    "# define the few shot examples\n",
    "fewshot_examples = [\n",
    "    {'input': 'I wrote about shakespeare',\n",
    "    'entities': [{'entity': 'I', 'time': 'present'}, {'entity': 'Shakespeare', 'time': '16th century'}],\n",
    "    'reasoning': 'I can write about Shakespeare because he lived in the past with respect to me.',\n",
    "    'answer': 'No'},\n",
    "    {'input': 'Shakespeare wrote about me',\n",
    "    'entities': [{'entity': 'Shakespeare', 'time': '16th century'}, {'entity': 'I', 'time': 'present'}],\n",
    "    'reasoning': 'Shakespeare cannot have written about me, because he died before I was born',\n",
    "    'answer': 'Yes'}\n",
    "]\n",
    "\n",
    "# Invoke the model\n",
    "generate = lm + anachronism_query(\"The T-Rex bit my dog\", fewshot_examples)\n",
    "\n",
    "# Show the extracted generations\n",
    "print(\"entities:\\n{0}\".format(generate['entities']))\n",
    "print(f\"reasoning: {generate['reason']}\")\n",
    "print(f\"answer: {generate['answer']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f3d0bb1-44c0-464b-8eb6-e9ef09c119e2",
   "metadata": {},
   "source": [
    "Now, let's run on the whole dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "530ae98a-d0c2-41f7-a6e0-80aebca6170a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 0px; padding: 0px; vertical-align: middle; padding-left: 8px; margin-left: -8px; border-radius: 0px; border-left: 1px solid rgba(127, 127, 127, 0.2); white-space: pre-wrap; font-family: ColfaxAI, Arial; font-size: 15px; line-height: 23px;'>Given a sentence tell me whether it contains an anachronism (i.e. whether it could have happened or\n",
       "not based on the time periods associated with the entities).\n",
       "\n",
       "Here are some examples:\n",
       "Sentence: I wrote about shakespeare\n",
       "Entities and dates:\n",
       "I : present\n",
       "Shakespeare : 16th century\n",
       "Reasoning: I can write about Shakespeare because he lived in the past with respect to me.\n",
       "Anachronism: No\n",
       "Sentence: Shakespeare wrote about me\n",
       "Entities and dates:\n",
       "Shakespeare : 16th century\n",
       "I : present\n",
       "Reasoning: Shakespeare cannot have written about me, because he died before I was born\n",
       "Anachronism: Yes\n",
       "\n",
       "Now determine whether the following is an anachronism:\n",
       "\n",
       "Sentence: Vasco de Gama avoided shipwreck by the Cape of Good Hope thanks to his astrolabe.\n",
       "Entities and dates:<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>V</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>as</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>co</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> de</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> G</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ama</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> :</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> </span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>1</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>5</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>th</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> century</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>A</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>st</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>rol</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>abe</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> :</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> </span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>1</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>5</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>th</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> century</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>C</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ape</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> of</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> Good</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> Hope</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> :</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> </span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>1</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>5</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>th</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> century</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>\n",
       "</span>Reasoning :<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> Vas</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>co</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> de</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> G</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ama</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> could</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> not</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> have</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> used</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> an</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> ast</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>rol</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>abe</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> to</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> avoid</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> ship</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>w</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>reck</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> because</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> the</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> ast</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>rol</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>abe</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> was</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> not</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> invent</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>ed</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> until</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> the</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> </span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>1</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>6</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>th</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> century</span><span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'>.</span>\n",
       "Anachronism:<span style='background-color: rgba(0.0, 165.0, 0, 0.15); border-radius: 3px;' title='1.0'> Yes</span></pre>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Few-shot Accuracy 0.41304347826086957\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "fews = []\n",
    "zero_shot = []\n",
    "count = 0\n",
    "for input, label in zip(inputs, labels):\n",
    "    print(f\"Working on item {count}\")\n",
    "    f = lm + anachronism_query(input, fewshot_examples)\n",
    "    f = 'Yes' if 'Yes' in f['answer'] else 'No'\n",
    "    fews.append(f)\n",
    "    count += 1\n",
    "fews = np.array(fews)\n",
    "\n",
    "\n",
    "print('Few-shot Accuracy', (np.array(labels) == fews).mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27d28fa3-4274-4db8-9fb3-c275615d0b29",
   "metadata": {},
   "source": [
    "Finally, we should shut down the server process we started:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cce6c032-7a41-490a-bc40-03a3a8df5521",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:     Shutting down\n",
      "INFO:     Waiting for application shutdown.\n",
      "INFO:     Application shutdown complete.\n",
      "INFO:     Finished server process [2026]\n"
     ]
    }
   ],
   "source": [
    "server_process.terminate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a409b16-4f87-4591-be69-fac4d4380727",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
